Node based path selection randomization

ABSTRACT

Methods and apparatus for randomizing selection of a next-hop path/link in a network are disclosed. An example method includes randomly selecting one or more path-selection randomization options to be applied to data packets processed in the network device. The example method further includes receiving a data packet and applying, by the network device, the one or more path-selection randomization operations to the data packet. The example method also includes determining a next-hop path for the data packet based on the one or more path-selection randomization operations and transmitting the data packet to a next-hop network device using the determined next-hop path.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit, under 35 U.S.C. §119, of U.S.Provisional Patent Application No. 61/225,538, U.S. Provisional PatentApplication No. 61/225,540 and U.S. Provisional Patent Application No.61/225,541, all filed Jul. 14, 2009. The disclosures of theseprovisional patent applications are incorporated herein by reference intheir entirety.

TECHNICAL FIELD

This description relates to data and network communications.

BACKGROUND

Network devices and data networks that process and communicate datatraffic often implement multiple paths across which data traffic may becommunicated In certain applications multiple paths, such as multiplepaths between a single source and single destination, may be groupedtogether (e.g., as a logical trunk or an aggregation group). In suchapplications, the network devices (nodes) communicating network datatraffic (e.g., data packets) may use load balancing techniques todistribute the data traffic across the links included in an aggregationgroup. One such technique is the use of hash-based load balancing.

To perform such hash-based load balancing across a set of aggregates, acommon approach is to hash a set of packet fields to resolve which amonga set of possible route choices to select (e.g., which member of anaggregate). At every hop in the network, each node may have more thanone possible next-hop/link that will lead to the same destination. Forpurposes of this disclosure, a node may be viewed as any level ofgranularity in a data network. For instance a node could be an incomingdata port a combination of the incoming data port and an aggregationgroup, a network device, a packet switch, or may be some other level ofganularity.

In a network or network device, each node would select a next-hop/linkbased on a hash of a set of packet fields which do not change for theduration of a flow. A flow may be defined by a number of differentparameters, such as source and destination addresses (e.g., IP addressesor MAC addresses), TCP flow parameters, or any set of parameters thatare common to a given set of data traffic. Using such an approach,packets within a flow, or set of flows that produce the same hash value,will follow the same path at every hop. This approach leads to poordistribution of multiple flows to aggregate members and causesstarvation of nodes, particularly in large multi-hop, multi-pathnetworks (e.g., certain nodes in a multi-hop network may not receive anydata traffic), especially as you move further away from the node (calledroot node) at which the traffic entered the network.

SUMMARY

A method and/or apparatus for path selection for data traffic, as setforth more completely in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a single-hop of a multi-pathnetwork in accordance with an example embodiment.

FIG. 2 is a block diagram illustrating two hops of a multi-path networkin accordance with an example embodiment.

FIG. 3 is block diagram illustrating a switching element in accordancewith an example embodiment.

FIGS. 4 a-4 c are destination mapping tables in accordance with exampleembodiments.

FIG. 5 is a flowchart illustrating a method for node-based pathselection randomization in accordance with an example embodiment.

FIG. 6 is a flowchart illustrating another method for node-based pathselection randomization in accordance with an example embodiment.

FIG. 7 is a block diagram illustrating an apparatus in accordance withan example embodiment that may be used to implement multi-stage hashingpath selection randomization.

FIG. 8 is a block diagram illustrating another apparatus in accordancewith an example embodiment that may be used to implement multi-stagehashing path selection randomization.

FIG. 9 is a flowchart illustrating a method for multi-stage hashing pathselection randomization in accordance with an example embodiment.

FIG. 10 is a flowchart illustrating another method for multi-stagehashing path selection randomization in accordance with an exampleembodiment.

FIG. 11 is a block diagram illustrating an apparatus in accordance withan example embodiment that may be used to implement parallel-stagehashing path selection randomization.

FIG. 12 is a flowchart illustrating a method for parallel-stage hashingpath selection randomization in accordance with an example embodiment.

DETAILED DESCRIPTION I. Multi-Hop Multi-Path Networks

FIG. 1 is block diagram illustrating a single-hop of a multi-pathnetwork 100 (network 100) in accordance with an example embodiment. Thenetwork 100 includes three nodes, Node0 105, Node1 110 and Node2 115. Inthe network 100, data traffic (e.g., data packets) may enter the network100 via Node 105. Depending on the data traffic, the Node0 105, afterreceiving the data traffic, may then select a next-hop/link for the datatraffic. In this example, the Node0 105 may decide to send certain datapackets to the Node1 110 and send other data packets to the Node2 115.These data packets may include data information, voice information,video information or any other type of information.

In a multi-path network, the Node1 110 and the Node2 115 may beconnected to other nodes in such a fashion that data traffic sent toeither node can arrive at the same destination. In such approaches, theprocess of binding a flow (such as discussed above) to a next-hop/linkmay begin by extracting a subset of static fields in a packet header(e.g., Source IP, Destination IP, etc.) to form a hash key. Typically,the hash key is fixed for a single flow and does not change for packetswithin the flow. If the hash key were to change for packets within aflow, a fixed binding of a flow to a next-hop/link would not beguaranteed and re-ordering of packets in that flow may occur at one ormore nodes of such a network. This packet re-ordering could lead todegraded performance for some communication protocols (e.g., TCP).

In such hash based path selection approaches, the hash key may serve asan input to a hash function, commonly a CRC16 variant or CRC32 variant,which produces, respectively, a 16-bit or 32-bit hash value. Typically,only a subset of the hash value bits is used by a given application(e.g., Trunking, link aggregation groups (LAGs) and equal costmulti-path (ECMP), herein, collectively, aggregation group(s)). Unusedbits of the hash value are masked out and only the masked hash value isused to bind a flow to one of the N aggregate members, where N is thenumber of links that belong to a given aggregation group.

The list of N aggregate members may be maintained in a destinationmapping table for a given aggregate. Examples of such destinationmapping tables is discussed in further detail below with respect to FIG.4 a. Each table entry contains forwarding information indicating a pathto the packet destination. An index into this table (a destination tableindex) may be calculated as the remainder of the masked hash valuemodulo N (the number of aggregate group members), such as the one shownbelow by Equation 1.destination table index=masked_hash_value mod N  Equation 1

Using the destination table index, the node may determine thenext-hop/link destination (aggregate member) for each packet. Thisprocess clearly binds a flow or set of flows producing the samedestination table index to a single aggregate member using amathematical transformation that will always select the same aggregatemember for a given hash key. As discussed in further detail below, thepath selection randomization techniques described herein may be used tomake such next-hop/link selections that also bind a given flow to aspecific aggregate member, but achieve better distribution of datatraffic across possible next-hops/link available at a given node.

As discussed above, the network 100 is a single-hop network (depth=1with two layers) that may be part of a larger multi-hop, multi-pathnetwork that performs forwarding for flows going to the same ordifferent destinations. As previously indicated, all data traffic thatis communicated in the network 100 traffic may enter the network 100 viaa single node (called root node), in this case, the Node0 105. Forpurposes of this example, it will be assumed that all flows can reachany destination of a larger network of which the network 100 is a partof using any leaf of an N-ary tree rooted at the Node0 105. In such anetwork, each flow originating at the Node0 105 will pick one member (ofthe N=2 aggregate members) to send the flow on its way to itsdestination using a hashing function. If each flow has a unique hash keythat is reasonably random as compared to other flows, and the hashfunction distributes hash-values equally over the hash values 16-bitspace, then flows arriving to the Node0 105 will be distributed evenlyto each of its two child nodes, Node1 110 and Node2 115 (e.g.,approximately fifty percent of the flows to each of Node1 100 and Node2115.

In this example, the network 100 is effectively a binary tree, and flowswould be distributed from the root node (Node0 105) to the two childnodes (Node1 110 and Node2 115). Given these conditions, flows areevenly distributed because there is only a single hop. Therefore, inthis example, neither Node1 110 or Node2 115 will receive adisproportionate number of flows and, accordingly, there are no starvedleaf nodes (i.e. leaf nodes that receive no traffic).

FIG. 2 is a block diagram illustrating two hops of a multi-path network200 in accordance with an example embodiment. As with the network 100discussed above, the network 200 may be part of a larger multi-hop,multi-path network that performs forwarding for flows going to same ordifferent destinations. Also in like fashion as discussed above withrespect to the network 100, all data traffic that is communicated in thenetwork 200 may enter the network 200 via a single node (called rootnode), in this case, the Node0 205. For purposes of this example, itwill be assumed that all flows can reach any destination of a largernetwork of which the network 200 may be a part of using any leaf of anN-ary tree (e.g., the larger network) rooted at the Node0 205.

By extending the depth of the network 100 to 2 and the number of layersto 3 (as compared with the network 100), both the Node1 205 and Node2210 will then have two children each. The Node1 205 will have childrenNode3 220 and Node4 225, as shown in FIG. 2. Similarly, Node2 215 willhave children Node5 230 and Node6 235.

In the network 200, if the same approach is used to determine hash keysand the same hash function is used for all nodes, an issue arises at thesecond layer of the network 200 as flows are received at Node1 210 andNode2 215. In this situation, each packet arriving at Node1 210 willyield the same hash key as Node0 205, when operating on the same subsetof packet fields (which is a common approach). Given the same hashfunction (e.g., a CRC16 hash function) and number of children, theresult of the hashing process at Node0 205 will be replicated at Node1210. Consequently, all flows that arrive at Node1 210 will be sent toNode3 220 as these are the same flows that went “left” at Node0 205.Because, in this arrangement, the same mathematical transformation (hashfunction) is performed on the same inputs (hash keys) at each node inthe network, the next-hop/link selected by the hash algorithm remainsunchanged at each hop. Thus, the next-hop/link selection between two ormore nodes in the flow path (e.g., Node0 205 and Node1 210) is highlycorrelated, which may lead to significant imbalance among nodes.

For a binary tree with a depth of 2 hops (three layers), the consequenceof this approach is that all flows that went “left” at the Node0 205 andarrived at the Node1 210 (e.g., all flows arriving at the Node1 210 fromNode0 205), will again go “left” at Node1 210 and arrive at Node3 220.As a result, Node4 225 will not receive any data traffic, thus leavingit starved. Similarly, all traffic sent to the Node2 215 will bepropagated “right” to the Node6 235, thereby starving the Node5 230. Asthe depth of such a network increases, this problem is exacerbated giventhat the number of leaf nodes increases (e.g., exponentially), but onlytwo nodes at each level will receive data traffic. By using the pathselection randomization techniques described herein, data traffic flowdistribution may be balanced at each node (hop), such that flowimbalances and the number of starved leaf nodes are reduced.

II. Example Switching Element

FIG. 3 is block diagram illustrating a switching element 300 inaccordance with an example embodiment. The switching element 300 may beused to implement the techniques for path selection randomizationdescribed herein. Also, other switching elements (e.g. switches,routers, packet processors, etc.) that perform forwarding or switchingdecisions for an aggregation group may be used to implement thesetechniques.

In the switching element 300, the forwarding/switching decisions madefor packets received via ingress ports 310 could be to select a next-hop(such as in the L3/ECMP forwarding decision logic 320 or the L2/ECMPforwarding decision logic 325) to which a packet should depart via theegress ports 315. Likewise, forwarding/switching decision made for thepackets received via ingress ports 310 may be to select a next link(such as a LAG or HiGig link (e.g., using the HiGig switching decisionlogic 330)) to which a packet should depart via the egress ports 315. Infact, multiple load balancing decisions may be performed in series, suchas depicted in FIG. 3. The path selection techniques described hereinmay be used, alone or in combination, for each forwarding or switchingdecision in a network in order to improve data traffic distributionacross members of an aggregation group and the network as a whole.

One advantage of the approaches described herein (e.g., when applied ina switching element 300 implemented in a multi-hop multi-path network)is that starvation of leaf nodes can be significantly reduced, if noteliminated, thus improving network utilization. Another advantage ofusing such techniques (e.g., in the switching element 300 or othernetwork device) is that, the distribution of flows for a wide range ofnetwork topologies may be improved when compared to a given networkusing the same hash key/hash function configuration for all nodes. Yetanother advantage of the techniques described herein, is that thesetechniques are network topology independent, and may be implemented in awide range of network topologies including networks within a network(e.g., where each node may operate as a network itself) to improve datatraffic distribution and network efficiency.

Yet another advantage of the techniques described herein, as has beenpreviously mentioned, is that nodes of a network (such as a multi-hopmulti-path network) can utilize any, or all, of the mechanisms presentedin this disclosure simultaneously to achieve performance gains.Independent configuration of the nodes of a network in accordance withthe approaches for path selection randomization described herein enableseach node of the network to perform different mathematicaltransformations at each node, or hop, (e.g., as compared to a previousnode, or hop) such that the output result in one node is much lesslikely to be repeated (except as consistent with random probability) inthe next or subsequent hops on the way to the destination.

While any, or all, of the techniques described herein may be utilized bya single node at the same time, attributes of the particular network inwhich the techniques are being applied may need to be considered inorder to time network performance. Therefore, certain techniques may bedisabled or eliminated in some networks. In general, however,implementing each of the techniques describe herein may provide anincremental improvement in performance for a wide range of networktopologies and data traffic scenarios.

For purpose of clarity and illustration, the following discussion oftechniques for path selection randomization are discussed in the contextof a multi-hop multi-path network (e.g., Trunking, LAG, ECMP, etc.). Itwill be appreciated, however, that these techniques may be applied inother network topologies in order to improve distribution of data flowsacross nodes of the network and, therefore, also improve networkutilization. Also, for purposes of clarity and illustration, thetechniques discussed herein may, in some instances, be described asbeing implemented together and, in other instances, described as beingimplemented independently. However, it will be appreciated that each ofthe techniques described herein may be implemented simultaneously or maybe implemented separately. Accordingly, the approaches described hereinare not limited to being implemented alone or in any particularcombination.

III. Node Based Path Selection Randomization

In order to improve path selection randomization between nodes of amulti-hop multi-path network, it would is desirable to have every nodeof a network perform a unique mathematical transformation on a uniquehash key, so that the result of each unique mathematical transformationhas no correlation with the result of the mathematical transformationsfrom all other nodes in the path of the flow from the root node to theleaf nodes of an N-ary tree network.

The following discussion discusses a number of techniques that, whenenabled simultaneously, may significantly increase path selectionrandomness at each node as compared to use of the same mathematicaltransformation at each node. These techniques include techniques formanipulating the hash key, hash function, hash output value, anddestination mapping table in a manner that does not lead to packetre-ordering.

When implementing these techniques, each node can configure logicentities of the node independently of all other nodes in the system. Forinstance, each node may implement logic entities (e.g., using machinereadable instructions) that allow a hash key, a hash function, a hashoutput value, and/or attributes of a destination mapping table to bemanipulated in a configurable manner.

Enabling these features reduces correlation between the forwarding nodesin a multi-hop multi-path network for next-hop/link (path) selection fordifferent flows. In doing so, imbalance may be reduced and nodestarvation, in a multi-hop network, may also be reduced.

a. Hash Key Seed and Bit Masking

One technique that may be used to increase path selection randomizationin a data network is for each node to randomly select operations formodifying a hash key before the hash key is used by a hash functionlogic entity to determine a hash value. As part of the networkconfiguration process, the randomly selected operations for each nodemay be analyzed and the techniques for one or more nodes may be modifiedin order to further reduce correlation between the nodes and increaserandomness. One of more of a number of various possible operations formodifying hash keys may selected by a node during an initialization ofthe nodes

In the example techniques discussed herein, the techniques that may berandomly selected for use at each node are techniques for modifying ahash key that is generated from fixed fields of a packet. By randomlyselecting one or more operation for modifying a hash seed (and thenapplying those options to each packet processed by a node) packet ordermay be preserved, because, for each node, the process of generating andmodifying hash seeds remains consistent throughout the duration of adata traffic flow.

One option for modifying a hash seed generated from a packet is toprepend, append or interleave a finite-sized seed value with the hashkey before passing it to the hash function for calculation of the hashvalue. The seed selected by each node may be of an arbitrary size. Inorder to improve path selection randomness but maintain packet order,each node in a network may be configured to have a different seed, but agiven node can not change its seed while it is actively transmittingpackets. If a node was to change its seed while actively transmittingpackets, it is likely the path selection decision for a given flow wouldchange and, as a result, packets for flows being processed by thenetwork may arrive at their destination out-of-order. In this situation,reordering of the packets may be needed. Such reordering may adverselyaffect network performance.

Another option for improving path selection randomness by modifying ahash seed is to mask selected hash key fields (bits) with a mask usingone or more logical “mask operations” using a mask value to produce amodified hash key. Such operations may include XORing the hash seed withthe selected mask value, ANDing the hash key with the selected maskvalue. ORing the hash key with selected mask value, and inverting (usinga logical NOT operation) the hash seed. Of course, other operation formasking a hash seed to produce a modified hash seed may be used. Thus,each node of a network may select a different seed to append/prepend to,and/or interleave (using logical operations) with, a hash key producedusing fixed packet fields or other fixed parameters associated with thepackets. The selected seed remains fixed for all flows for a given node,even though each node may utilize different seeds.

Also, hash key fields may be optionally masked using any bitwiseoperation (XOR, AND, OR, etc.) with a randomly chosen per node mask, aswell as a randomly chosen per node masking operation. In order topreserve packet ordering, the mask and the masking operation would beselected at initialization time for a given node and not changed whilethe node is transmitting data packets, so that the same mask and maskingoperation is applied to all packets in a given flow. However, multiplenodes should use different masks and masking operations in order toincrease path selection randomness.

Accordingly, when using such techniques, each node should randomlyselect a unique seed, mask and masking operation at configuration time.Once configured, the seed and mask must remain fixed throughout theduration of operation. By incorporating a seed with a hash key andmasking the hash key with a mask operation reduces the correlation innext-hop/link selection for successive hops, which leads to reducedimbalance in path selection and, therefore, better distribution of datatraffic to leaf nodes in a multi-hop multi-path network.

b. Hash Function Selection

Once the hash key is generated and modified using one or moremodification operations, such as those discussed above, the modifiedhash key may be provided as an input to a hash function. A logicalentity in a network node that is used to implement the hash function mayreceive the modified hash key and calculate the hash value using aselected hash function for the node. Each node may randomly select thehash function that it uses to determine hash values from modified hashkeys. The hash function implemented by a given node may one of a CRC16hash function, a CRC32 hash function, a Pearson's hash function or anXOR hash function. Of course, a given node may have other hash functionsto select from and may implement a number of other hash functionsinstead of the examples listed above.

In order to increase path selection randomness, it would be desirable tohave each node use a unique hash function, as such an approach wouldsignificantly minimize correlation in the next-hop/link selection.However, in certain embodiments, such an approach may impractical.Network devices (nodes) that implement hash function path selection mayonly provide a small set of hash functions to select from.

As with the hash key modification operations discussed above, the hashfunction implemented by a given node should be chosen at random fromthose available during configuration, but remain fixed while the node istransmitting packets in order to preserve packet ordering. By randomlyselecting which of a set of available hash functions to use at each nodeof a network, after successive hops of data packet, where the hashfunction at each node is different, the correlation between pathselection for a given flow at each node is reduced and flows should beevenly distributed as they traverse down the network tree. Again, thisapproach leads to increased path selection randomness and balanceddistribution of flows to the leaf nodes.

c. Hash Value Bit Selection

By applying a selected hash function implemented at a given node to amodified hash key produced by the node, a finite-sized (typically)16-bit or 32-bit hash value is produced. This hash value may be used foruse by load balancing applications (e.g. Thinking, LAGs, and ECMPs),including next-hop/link selection. For instance, the hash value may beused to compute a destination table index, such as previously describedand discussed in further detail below. For implementation reasons incertain embodiments, only a subset (k) bits of the hash value may beused to compute the destination table index. The k bits used to computethe destination table index may also be referred to as a selected hashvalue.

For instance, a given network node may utilize only 8 bits of a 16-bithash value to resolve a destination table index. Consequently, in suchan approach, some number of hash value bits, in this case 8-bits, gounused by the given node. To reduce correlation of next-hop/link outputvalues among nodes, an individual node may be configured to select a setof k bits at random (which are randomly determined at initialization)from the hash values generated by the node to resolve destination tableindices for the given node. Different nodes in a network can select adifferent random set of k bits from the hash value. In order to preservepacket ordering, the set of k bits used for a given node should remainfixed while the node is transmitting packets. This increased randomnessreduces imbalance at each node in the network and improves flowdistribution to leaf nodes. In order to further increase randomness,each node may choose to use a random value for k. Thus, a given node mayselect a different number of bits than other nodes to generate selectedhash values.

d. Destination Mapping Table Shuffling and Entry Replication

After a hash value is generated and a selected hash value is determinedfrom the hash value, such as in the fashions discussed above, anext-hop/link selection for an associated data packet may be determinedby computing an index into a Destination Mapping Table (e.g., adestination mapping index) and then indexing the destination mappingtable with the computed index. In an example embodiment, a destinationmapping index may be the remainder of the selected hash value modulo N,where N is the number of members in a given aggregation group.Randomization of the destination mapping table index and subsequentnext-hop/link selection may be achieved by determining the selected hashvalue used to determine the destination mapping table index in themanner discussed above.

In order to further increase path selection randomization, entryreplication may be used for entries in a destination mapping table, suchas discussed further below with respect to FIGS. 4 b and 4 c. Using suchan approach, each entry in a given destination mapping table may bereplicated by a factor “rf.” The rf factor may be selected randomly byeach node in a network for each associated aggregate that is availableas a next-hop/link. The rf factors should be selected by the nodesduring initialization and should remain fixed for a given node while thegiven node is transmitting packets, or until changes are required to theaggregate's membership, so as to preserve packet ordering. Using entryreplication alters the definition of the destination mapping table indexto the calculation shown by Equation 2 below:Destination Mapping Table Index=selected hash valuemodulo(rf*N)  Equation 2Using such an approach, each next hop/link that may be selected by agiven node will appear rf times in an associated destination mappingtable. For example, a node with four child nodes that may be selected asa next-hop/link, where the node is using a replication factor of 4 willhave 16 entries in its destination mapping table (e.g., with indices of0-15) and each child node will be associated with 4 of the 16 indices.

In addition to replicating table entries, entries in a destinationmapping table may be shuffled to further increase path selectionrandomization. In a network that uses entry shuffling, the order inwhich next-hops/links appear in the destination mapping table for agiven node is randomized by the given node. Such shuffling should beperformed at configuration and should remain fixed throughout theduration of operation or until changes are required to the aggregate'smembership, so as to preserve packet ordering.

Such entry shuffling modifies the next-hop/link selection sequence. Themodulo operator splits the range of values provided by the selected hashvalue into rf*N distinct sets of numbers. With shuffling, rf randomindices map to the same next-hop/link at any node. If any two nodes (saynode-x and node-y) that are in the path of a set of flows have the sameconfiguration of other parameters and shuffling is not performed, andthe flows traverse node-x before they traverse node-y, all flows thatmapped to the same next-hop/link at node-x will also map to onenext-hop/link at node-y. Such a result leads to poor flow distributionat node-y and may result in disproportionate distribution of datatraffic. By using replication and shuffling, the probability of thisoccurrence is reduced.

While shuffling can be implemented without the use of replication, usingreplication in combination with shuffling allows for an increased numberof possible shuffling permutations. For instance, each aggregation groupfor a given node can be configured using a unique replication factor. Asthe replication factor increases for a given node, the range of valuesprovided by computing hash value modulo N will provide rf*N distinctsets of numbers. Each set can be mapped to a single next-hop/linkrandomly (a.k.a., shuffling) at configuration time. This reduces thecorrelation among next-hop/link sequences produced at each node, whichserves to reduce imbalance and starvation in a multi-hop network.

e. Example Destination Mapping Tables

FIGS. 4 a-4 c are example destination mapping tables that may be used bya node in a multi-hop multi-path network, where the node connects tofour aggregate members (e.g., next-hops/links) of an aggregation group,where the members are designated {a, b, c, d}. FIG. 4 a is a destinationmapping table 400 that may be used in a node that does not implementreplication or shuffling, as were described above. As may be seen inFIG. 4 a, each member {a, b, c, d} of the aggregation group has a singleentry in the destination mapping table, and the aggregate members arelisted in sequential order in the table 400's entries.

In order to index the table 400 for a given packet, a selected hashvalue is generated to use in determining an index into the table 400.This index may be determined using Equation 3 presented below:Destination Mapping Table Index=selected hash value modulo 4  Equation 3

The modulo operator in Equation 3 corresponds with (is equal to) thenumber of entries in the destination mapping table, which is four inthis situation. In an embodiment that generates a selected hash valuethat is 8-bits, there are at most 256 possible unique hash values. Themodulo operation shown in Equation 3 will then map each of the 256unique values into 1 of 4 index values uniformly. Selected hash valuesof (0, 4, 8, . . . , 252) will all map to index 0. Similar (count byfour) sequences of selected hash values will map, respectively, toindices 1, 2, and 3. The destination (next-hop/link) is then determinedby using the determined index to index the destination mapping table. Ifeach node in a multi-hop multi-path network produces the same selectedhash value for many different flows and those hash values produce thesame index, then those flows may be assigned to the same child node,which may result in an uneven flow distribution.

As was discussed above, correlation between path selection fornext-hops/links can be reduced by performing entry replication and entryshuffling for a destination mapping table. FIG. 4 b illustrates adestination mapping table 410 that uses a replication factor of threefor the node discussed above that is connected with the aggregate groupincluding members {a, b, c, d}. In this situation, each entry in thedestination mapping table 410 is replicated three times, as is shown inFIG. 4 b, where each aggregate member has three entries in the table410.

For each packet arriving at the node of this example, a selected hashvalue may be generated to use in determining an index into thedestination mapping table 410. Given the table is now larger by a factor(rf) of the three (as compared to the table 400), the index for thetable 410 may be determined using Equation 4 presented below:Destination Mapping Table Index=selected hash value modulo 12  Equation4The modulo operation of Equation 4 maps each of the 256 possibleselected hash values into 1 of 12 index values uniformly.

As shown by the destination mapping table 420 illustrated in FIG. 4 c,the sequence of the aggregate members associated with a node may berandomized at each node by shuffling the order in which entries appear(e.g., such as compared to the table 410). Implementing shuffling andreplication at each node of a multi-hop multi-path network randomizesthe destination output sequences produced at each node and increases theindex space into which selected hash values map. Such techniques mayincrease path flow selection randomness and thus reduce data trafficdistribution imbalances in an associated multi-hop multi-path network.

f. Example Methods

FIG. 5 is a flowchart illustrating a method 500 for node-based pathselection randomization in accordance with an example embodiment. Themethod 500 may be implemented, for example, in the network 100 or thenetwork 200, where any or all of the nodes of those networks mayindividually implement the method 500. Further, the method 500 may beimplemented in the switch element 300 shown in FIG. 3, such as by any,or all of the indicated load balancing points. Of course, the method 500may implemented in a number of other network devices, such as switchesand routers, for example.

The method 500 includes, at block 505, initializing a network device byrandomly selecting one or more path-selection randomization options tobe applied to data packets processed by the network device. Thepath-selection randomization options may include one or more of thetechniques described above. For example, the randomization options mayinclude combining a hash key with a hash seed, such as in the mannersdiscussed above. Of course, any, or all of the techniques describedherein, as well as other path selection randomization techniques may beselected at block 505.

The method 500 further includes, at block 510, receiving a data packet.The data packet may be received via, for example, an ingress port of anetwork device. The data packet may, however, be received from a numberof other sources. The method 500 also includes, at block 515 applyingthe selected path randomization options to the packet. As discussedabove with respect to block 505, applying the one or more pathrandomization operations to the packet may include applying any, or allof the techniques discussed above. For example, applying the pathselection randomization options may include determining a selected hashvalue from a received hash value, masking a hash key and selecting ahash function.

The method 500, also includes, at block 520, determining a next hop forthe data packet based on the one or more path selection randomizationoptions selected at block 505 and applied at block 515. For instance,the next hop may be determined using a destination mapping index tablethat is implemented using the replication and shuffling techniquesdiscussed above. Of course, any of the approaches for path selectionrandomization described herein may be implemented in accordance with themethod 500 shown. The method 500 still further includes, at block 525,transmitting the data packet to a next-hop network device using the nexthop path determined at block 520.

FIG. 6 is a flowchart illustrating another method 600 for node-basedpath selection randomization in accordance with an example embodiment.As with the method 500, the method 600 may be implemented, for example,in the network 100 or the network 200, where any, or all of the nodes ofthose networks may implement the method 600. Further, the method 600 maybe implemented in the switch element 300 shown in FIG. 3 at any, or allof the indicated load balancing points. Of course, the method 600 mayimplemented in a number of other network devices, such as switches androuters, for example.

In the method 600, blocks 605, 610 and 615 may be implemented in likefashion as the corresponding blocks 505, 510 and 515 of the method 500illustrated in FIG. 5. Therefore, for purposes of brevity, those blocksare not described in detail again here.

At block 620, the method 600 includes determining a hash value for thedata packet. The hash value at block 620 may be a selected hash valueand may be determined using any, or all of the techniques fordetermining a hash value. Of course, the hash function may be determinedin a number of ways at block 620.

The method 600 further includes, at block 625, determining an index fora destination mapping table. The index may be determined, for example,by using one of the Equations 2-4, such as described above. Forinstance, a table using replication and/or shuffling may be used, as wasdiscussed above and as illustrated in FIG. 4 c. At block 630, the method600 still further includes indexing the destination mapping table usingthe determined index to determine a next-hop path for the data packet.For the method 600, the index determined at block 630 may be used todetermine a next-hop path using a destination mapping table, such as oneof the tables 400, 410 and 420 shown in FIGS. 4 a-4 c, for example. Themethod 600 still further includes, at block 635, transmitting the datapacket to a next-hop network device using the next hop path determinedat block 630.

IV. Multistage Path Selection Randomization

As was discussed above, a number of different parameters, such as sourceand/or destination addresses (e.g., IP addresses or MAC addresses),portions of the source and/or destination addresses. TCP flowparameters, or any set of parameters that are common to a given set ofdata traffic may define a flow. Flows may be defined differently at eachnode and may even be defined differently by different portions of asingle node. As was also discussed above, balanced flow distribution maybe achieved by adding flow-based randomization components to the loadbalancing mechanism at each node.

Another approach that may be implemented, alone or in combination withone or more of the other techniques described herein, is the use ofmulti-stage, or cascading, hash functions. Using this approach, multiplehash functions can be implemented in a cascaded fashion in an attempt torandomize next-hop/link selection on a per-flow basis.

Another technique for increasing path selection randomization, inaddition to those already discussed and those discussed below, isflow-based hash configuration, which may be implemented in conjunctionwith multi-stage or cascading hash function as described herein. Thisapproach enables a flow balancer to be configured uniquely, on aper-flow or per-set-of-flows basis, at each node of a multi-hopmulti-path network. In fact, different nodes can chose differentconfigurations for the same flow as long as a node consistently uses thesame configuration throughout the duration of any flows it istransmitting, which preserves packet order. The following discussiondescribes these two techniques in further detail.

a. Multi-Stage/Cascading Hash Function Path Selection

FIG. 7 is a block diagram illustrating an apparatus 700 in accordancewith an example embodiment that may be used to implement multi-stagehash path selection randomization. The apparatus 700 usesmulti-stage/cascading hash functions to increase the entropy, orrandomness, in generating a hash value that is used for next-hop/linkselection in order to distribute data traffic flows more evenly acrossavailable paths in multi-hop multi-path networks.

The apparatus 700 is implemented using 2-stages. As shown in FIG. 7, theapparatus 700 includes a first hash key generator 705, a hash functionF₁ stage 710, a second hash key generator 715 and a second hash functionF₂ stage 720. The hash key generation modules 705 and 715 may use thetechniques for modifying a hash key that were discussed above togenerate the hash keys K₁ and K₂, in addition to using the specifictechniques for hash key generation that are discussed with respect toFIG. 7.

As shown in FIG. 7, the hash key generator 705 generates a first hashkey, K₁. The hash key K₁ may, for example, be generated using a hash keygenerated from fixed fields of a packet and then modifying the hash keygenerated from the fixed packet fields using the techniques discussedabove. As is also shown in FIG. 7, the first stage hash function F₁stage 710 may compute a hash value, designated V₁, using the hash key,K₁, the hash value V₁ may be computed using the techniques describedherein, such as randomly selecting a hash function to use at a givennode.

For the second stage hash function F₂ stage 720, the hash key, K₂, thatis input to the second hash function F₂ stage 720 could be generated byprepending V₁ to K₁, which may be done by the hash key generator 715.Using the hash output value V₁ from stage 1 rather than a random,non-faxed seed is important, as the hash value V₁ is the same for allpackets in a flow, which prevents re-ordering when prepending it withthe first hash key K₁ and it is unique to a given flow, so also improvesrandomness in the hash values V₂ produced across multiple flows.

Prepending the value V₁ to a hash key input to the second hash functionstage 720 provides many more bits of “random” information as an inputfor generating a hash value V₂, which then translates to more randomnessin the output hash value V₂. This increased randomness in the outputhash value V₂ can reduce imbalance and starvation of nodes in largemulti-hop networks when the hash value V₂ is used to determinenext-hops/links for data flows being transmitted by a node of amulti-hop multi-path network, such as using the techniques describedabove.

In addition to prepending the first stage 710 hash value output V₁ to K₁to produce the second hash key K₂, the output V₁ could also be used toreplace select bit fields in the hash key K₂ used by the second hashfunction stage 720. Using this approach, the hash key K₂ may begenerated directly from fixed data packet fields. In this example, ifthe bit fields in the hash key K₂ selected for replacement are fixedacross multiple flows, then this approach may introduce additionalrandomization into the hashing process. Generally speaking, anyarbitrary function that generates a distinct hash key from anycombination of (1) the output of a hash function, (2) a hash key, (3) ahash seed, or (4) packet fields to yield a distinct hash key for aseparate hash function may be used to generate the hash keys K₁ and K₂used in the apparatus 700. Additionally, the techniques discussed abovefor modifying a hash key, such as prepending, appending or interleavinga random seed selected at initialization or masking a hash key usingmasked logical or bitwise operations may be used to generate, at leastin part, one or both of the hash keys K₁ and K₂ used in the apparatus700.

b. Flow-Based Hash Configuration

Configuration of hash-based load balancing techniques is generally,performed upon powering on a network device implementing such techniquesand then left static for the duration of time that the device operatesand is processing data traffic. This approach is taken since theperformance of hash-based load balancing is sensitive to packetre-ordering. Such re-ordering is prevented by ensuring all packetswithin a flow hash to the same value (and select the same path) bykeeping the hash configuration static during continuous operation. Thefollowing discussion describes a technique for hash based load balancingthat maintains packet order while also allowing the hash configurationto change dynamically by binding a hash configuration to a flow or setof flows.

In an example embodiment, binding a hash configuration to a flow, or setof flows, is achieved using a Flow Configuration Mapping Table thatreceives, as its input, a set of bits that are unique to the flow. Thesebits can be either the packet fields that form the hash key or, ifmulti-stage hashing is performed, the hash value from a prior stage, astwo examples. For the purpose of this discussion, the set of bits usedto identify the flow and to index the Flow Configuration Mapping Tableis referred to as the FlowID. The FlowID can be made up of select packetfields, the hash value, or other information that can be used touniquely identify a flow or set of flows. In this example, the FlowConfiguration Mapping Table (FCMT) utilizes the FlowID as an index intothe table.

In the example embodiment, each FCMT entry contains information thatdefines how the load balancer should be configured for a given flow ormicroflow. Configuration options can include defining the hash functionto utilize, which set of bits from the hash value to use when performingnext-hop/link selection, or how to use each of the hash outputs whenutilizing a multi-stage hash. A wide range of configuration options areavailable with some possible techniques discussed above.

The benefits provided by such an FCMT are more apparent when implementedin a multi-hop multi-path network, because the flow balancing functionperformed at each node can be unique to the per-flow or per-set-of-flowsgranularity. This reduces correlation in the next-hop/link selectionprocess between nodes which, in turn, reduces next-hop/link imbalanceand starvation/overutilization of links and nodes.

FIG. 8 is a block diagram illustrating an apparatus 800 in accordancewith an example embodiment, where Flow Based Configuration isimplemented in conjunction with the multi-stage hashing path selectionrandomization apparatus 700 shown in FIG. 7. The elements of theapparatus 700 are referenced with the same reference numbers as in FIG.7. The elements of the apparatus 700, for the sake of brevity will notbe described in detail again here, except to explain the flow basedconfiguration aspects of the apparatus 800.

In the apparatus 800, the hash configuration is determined using a FlowConfiguration Mapping Table 810. While the information in the FlowConfiguration Mapping Table 810 in the apparatus 800 is shown in onetable, in other embodiments, the flow configuration mapping informationfor each stage could be implemented in a separate, respective table.

In the apparatus 800 shown in FIG. 8, the lower two bytes of a Source IP(SIP) are used to determine the index (FlowID) for each packet. The hashconfiguration for the flow can then be determined using the FlowConfiguration Mapping Table 810 entry corresponding to the identifiedindex. For example, for all packets having a SIP whose lower two bytesare 36.159, the apparatus 800 would be configured such that hashfunction #1 implemented at hash function stage 710 is CRC16-A, hashfunction #2 implemented at hash function stage 720 is CRC32-A, with ahash value offset of 13, and will mask using an XOR operation. For allpackets having a SIP whose lower two bytes are 59.122, apparatus 800will be configured such that hash function #1 at hash function stage 710is CRC32-B, hash function #2 at hash function stage 720 is CRC16-A, witha hash value offset of 7, and will mask using an OR operation.

c. Example Methods

FIG. 9 is a flowchart illustrating a method for multi-stage hashing pathselection randomization in accordance with an example embodiment. Themethod 900, at block 905, may include performing an initializationsequence in a network device. The initialization sequence of block 905may include generating one or more flow mapping configuration tables,such as the FCMT 810 discussed above. In the method 900, the FCMT mayinclude a plurality of indices. Each of the plurality of indices of theFCMT may have associated with it in the FCMT: one or more respectiveoperations for generating a first hash key; a respective first hashfunction; one or more respective operations for generating a second hashkey; a respective second hash function; and a respective offset.

The method 900, at block 910, further includes receiving a data packet.At block 915, the method 900 includes indexing, based on the datapacket, the one or more flow configuration mapping tables toselect/determine path selection randomization options based on the FCMT.In the method 900, the selected path based randomization options mayinclude one or more operations for generating a first hash key; a firsthash function; one or more operations for generating a second hash key;a second hash function; and an offset.

The method 900, at block 920, includes generating the first hash keybased on the data packet. One or more operations for generating thefirst hash key selected at block 915 may be used to generate the hashkey at block 920. At block 925, the method 900 includes generating thefirst hash value from the first hash key using the first hash function.The first hash function selected at block 915 may be used at block 925.

The method 900 at block 930, includes generating the second hash keybased on at least two of the data packet, the first hash value and theoffset. Also, one or more operations for generating the second hash keyselected at block 915 may be used to generate the hash key at block 930.At block 935, the method 900 includes generating the second hash valuefrom the second hash key using the second hash function. The second hashfunction selected at block 915 may be used at block 935. The method 900,at block 940, then includes selecting a next-hop path based on thesecond hash value.

As has been previously discussed, the various approaches describedherein may be implemented simultaneously. Also, the various approachesfor determining a FlowID (index) of a FCMT, selecting a hash function,selecting operations to modify hash keys and selecting operations formodifying an output hash value may be applied in conjunction with theoperations of the method 900 (and the other methods and apparatusdescribed herein) as appropriate.

FIG. 10 is a flowchart illustrating another method 1000 for multi-stagehashing path selection randomization in accordance with an exampleembodiment. The method 1000, at block 1005, may include receiving, at anetwork device, a data packet. The method 1000 may also include, atblock 1010, generating a first hash key based on the data packet and, atblock 1015, generating a first hash value from the first hash key usinga first hash function.

The method 1000 may still further include, at block 1020, generating asecond hash key based on the data packet and the first hash value and,at block 1025, generating a second hash value from the second hash keyusing a second hash function. The example method may also furtherinclude, at block 1030, generating a third hash key based on the datapacket and at least one of the first hash and the second hash value,and, at block 1035, generating a third hash value from the third hashkey using a third hash function. The example method may also furtherinclude, at block 1040, selecting a next-hop path/link based on thethird hash value. In other embodiments, an arbitrary number of hash keysand hash functions may be used. Also, the method 900 may be implementedusing any of the approaches for the various operations described herein,as well as being combined with other techniques described herein, asappropriate.

A possible variation on the FCMT techniques described in this disclosureis to maintain additional information to identify a given flow (i.e.Flow ID) that is not discussed herein. Another possible variation may beto use a look up table to produce a unique value for use innext-hop/link selection, rather than use a traditional hash function.

V. Parallel Hash Function Randomization

FIG. 11 is a block diagram illustrating an apparatus 1100 in accordancewith an example embodiment that may be used to implement parallel-stagehashing path selection randomization. The apparatus 1110 includes a hashkey generator 1105. The hash key generator 1105 may generate multiplehash keys using a number of techniques, including the techniques forhash key generation and modification that are described herein.

The apparatus 1100 also includes three parallel hash function blocks1110, 1115 and 1120. The parallel hash function blocks 1110, 1115 and1120 receive hash keys (respectively K₁, K₂ and K₃) from the hash keygenerator 1105. The hash function blocks 1110, 1115 and 1120 thengenerate hash values (respectively V₁, V₂, and V₃) based on the hashkeys K₁, K₂ and K₃. As previously discussed herein, the randomizationtechniques described herein may be applied to the blocks of theapparatus 1100 as appropriate. For instance the hash functions for eachof the hash function blocks 1110, 1115 and 1120 may be randomlyselected, such as in the manners described above, or in some otherfashion.

The hash values V₁, V₂ and V₃ may then be provided to a hash valuegenerator 1125, which generates a hash value output value V_(f) based onthe three parallel produced hash values V₁, V₂ and V₃. As the apparatus1100 may operate using any of the techniques disclosed herein, asappropriate, in one embodiment, the hash key generator 1105 may, atinitialization, select one or more operations that are used to modify ahash seed that only includes fixed fields of an incoming data packet.The operations may be the same for each of the hash keys generated or,alternatively, may be different operations. Of course other techniques,such as those described herein, may be implemented in the apparatus1100.

The hash value generator 1125 may use a number of techniques forcombining the hash values V₁, V₂ and V₃ received from the parallel hashfunction blocks 1110, 1115 and 1120. For instance the output values V₁,V₂ and V₃ of the of the multiple parallel hash function blocks 1110,1115 and 1120 may be combined to produce a single hash value (V_(f)),using a number of techniques, such as those discussed herein. Forinstance, as one example, the hash values V₁, V₂ and V₃ may be combinedto produce the hash value V_(f) using an XOR function. In such anapproach, the hash value V_(f) may be produced by performing an XORfunction with the three hash values V₁, V₂ and V₃ as inputs and V_(f) asthe output. For example, in this situation, the hash value V_(f) may bedetermined by the logical expression V_(f)=(V₁) XOR (V₂) XOR (V₃).

Another approach may be to interleave the hash values V₁, V₂ and V₃ toproduce the hash value V_(f) using the logical expressionV_(f)=concat(V₁[15:8], V₂[7:4], V₃[3:0]), where the bits positions ofthe interleaved segments of each input hash value match their originalbit positions before concatenation. Still another approach would be toproduce the hash value V_(f) by interleaving segments of the hash valuesV₁, V₂ and V₃ but shuffling the order of the segments so that theoriginal bit positions from the hash values V₁, V₂ and V₃ do not matchthe bit positions in the hash value V_(f). Such an interleaving withshuffling operation may be implemented using a logical expression, suchas V_(f)=concat(V₁[7:0], V₂[11:8], V₃[16:12]), for example.

FIG. 12 is a flowchart illustrating a method 1200 for parallel-stagehashing path selection randomization in accordance with an exampleembodiment. The method 1200 may include, at blocks 1205-1230 performing,by the network device, an initialization sequence. The initializationsequence may include, at block 1205, selecting one or more operationsfor generating a first hash key and, at block 1210, selecting the firsthash function from a plurality of hash functions. The initializationsequence, at block 1215, may also include selecting operations forgenerating a second hash key and, at block 1220, selecting the secondhash function from the plurality of hash functions. The initializationsequence may still further include, at block 1225, selecting operationsfor generating a third hash key and, at block 1230, selecting the thirdhash function from the plurality of hash functions.

At block 1235, the method 1200 may include receiving, e.g., at thenetwork device, a data packet and, at block 1240, generating a firsthash key based on the select packet fields. The method 1200 may alsoinclude, at block 1245, generating a first hash value from the firsthash key using a first hash function. The example method may stillfurther include, at block 1250, generating a second hash key based onthe data packet (and modification operations) and, at block 1255,generating a second hash value from the second hash key using a secondhash function (e.g., in parallel with generating the first hash key andgenerating the first hash value).

The example method may further include, at block 1260, generating athird hash key based on select packet fields and, at block 1265,generating a third hash value from the third hash key using a third hashfunction (e.g., in parallel with generating the first hash key,generating the first hash value, generating the second hash key andgenerating the second hash value). As shown in FIG. 12, the method 1200may still further include, at block 1270, combining the first hashvalue, the second hash value and the third hash value to produce acombined hash value and, at block 1275, selecting a next-hop/link basedon the combined hash value. The next-hop/link may be selected, forexample, using techniques described herein.

In such example method, the first hash function, the second hashfunction and the third hash function may be different hash functions.Further, in such example methods, the first hash key, the second hashkey and the third hash key may be different hash keys.

In the example method, combining the first hash value, the second hashvalue and the third hash value may include combining the first hashvalue, the second hash value and the third hash value using an arbitraryfunction. The arbitrary function may be one of an XOR function, and ORfunction, an AND function and a NAND function, or any combination ofthese functions and other functions, such as mathematical functions.

Alternatively, in the example methods, combining the first hash value,the second hash value and the third hash value may include interleavingbits of the first hash value, bits of the second hash value and bits ofthe third hash value. In the combined hash value, the interleaved bitsof the first hash value, the interleaved bits of the second hash and theinterleaved bits of the third hash value may occupy same bit positionsas compared to their respective bit positions in the first hash value,the second hash value and the third hash value. Of course, fewer or morehash values may be generated and/or interleaved/combined.

As another alternative, in the example methods, combining the first hashvalue, the second hash value and the third hash value may compriseinterleaving bits of the first hash value, bits of the second hash valueand bits of the third hash value. In the combined hash value, theinterleaved bits of the first hash value, the interleaved bits of thesecond hash value and the interleaved bits of the third hash value mayoccupy different bit positions as compared to their respective bitpositions in the first hash value, the second hash value and the thirdhash value. Again, fewer or more hash values may be generated and/orinterleaved/combined. A possible variations on the parallel hash pathselection techniques described in this disclosure may utilize a logicaloperation (AND, OR, etc.) for combining hash outputs not listed in thedescription.

Implementations of the various techniques described herein may beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. Implementations mayimplemented as a computer program product, i.e., a computer programtangibly embodied in an information carrier, e.g., in a machine-readablestorage device, for execution by, or to control the operation of, dataprocessing apparatus, e.g., a programmable processor, a computer, ormultiple computers. A computer program, such as the computer program(s)described above, can be written in any form of programming language,including compiled or interpreted languages, and can be deployed in anyform, including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program can be deployed to be executed on one computer or onmultiple computers at one site or distributed across multiple sites andinterconnected by a communication network.

Method steps may be performed by one or more programmable processorsexecuting a computer program to perform functions by operating on inputdata and generating output. Method steps also may be performed by, andan apparatus may be implemented as, special purpose logic circuitry,e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor for executing instructions and one or more memorydevices for storing instructions and data. Generally, a computer alsomay include, or be operatively coupled to receive data from or transferdata to, or both, one or more mass storage devices for storing data,e.g., magnetic, magneto-optical disks, or optical disks. Informationcarriers suitable for embodying computer program instructions and datainclude all forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in special purposelogic circuitry.

To provide for interaction with a user, implementations may beimplemented on a computer having a display device, e.g., a cathode raytube (CRT) or liquid crystal display (LCD) monitor, for displayinginformation to the user and a keyboard and a pointing device, e.g., amouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

Implementations may be implemented in a computing system that includes aback-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user can interact with animplementation, or any combination of such back-end, middleware, orfront-end components. Components may be interconnected by any form ormedium of digital data communication, e.g., a communication network.Examples of communication networks include a local area network (LAN)and a wide area network (WAN), e.g., the Internet.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the embodiments of the invention.

What is claimed is:
 1. A method for processing packet data in a networkdevice, the method comprising: randomly selecting, by the network deviceone or more path-selection randomization options to be applied to datapackets processed in the network device; receiving, at the networkdevice a data packet: applying, by the network device, the one or morepath-selection randomization operations to the data packet; determininga next-hop path for the data packet based on the one or morepath-selection randomization operations; and transmitting, from thenetwork device, the data packet to a next-hop network device using thedetermined next-hop path, wherein the path-selection randomizationoptions comprise operations for modifying a hash key, the operations formodifying a hash key including 1) operations for prepending a first hashkey seed to respective hash keys of received data packets; 2) operationsfor appending a second hash key seed to the respective hash keys of thereceived data packets; and 3) operations for masking randomly selectedbits of the respective hash keys of the received data packets.
 2. Themethod of claim 1, wherein the path-selection randomization optionscomprise: operations for modifying a hash key; a plurality of hashfunctions; and operations for modifying a hash function output value. 3.The method of claim 1, the operations for modifying a hash key furtherincluding operations for interleaving a third hash key seed to therespective hash keys of the received data packets.
 4. The method ofclaim 3, wherein the operations for masking randomly selected bitsinclude one or more of a logical XOR operation, a logical AND operation,a logical OR operation and a logical NOT operation.
 5. The method ofclaim 1, wherein randomly selecting one or more path-selectionrandomization options includes selecting one of a plurality of hashfunctions.
 6. The method of claim 5, wherein the plurality of hashfunctions includes one or more of a cyclic redundancy check (CRC)16 hashfunction, a CRC32 hash function, an exclusive or (XOR) hash function anda Pearson's hash function.
 7. The method of claim 1, wherein thepath-selection randomization options comprise operations for modifying ahash function output value, the operations for modifying the hash valuecomprising operations to select a random set of bits from respectivehash values of the received data packets, the random set of bits beingselected during an initialization process.
 8. The method of claim 1,further comprising: determining, by the network device, a hash value forthe received data packet based on the selected path-selectionrandomization options; determining, based on the hash value, an indexfor a destination mapping table; and indexing the destination mappingtable using the determined index to determine the next-hop path for thedata packet.
 9. The method of claim 8, wherein determining the index forthe destination mapping table and indexing the destination mapping tablecomprises: determining a modulo of the hash value based on a number ofavailable next-hop paths N and a replication factor rf a correspondingaggregate of the network device, wherein the destination mapping tableincludes rf entries for each respective next-hop path for thecorresponding aggregate.
 10. The method of claim 9, wherein thereplication factor rf is randomly selected during an initialization forthe corresponding aggregate.
 11. The method of claim 1, furthercomprising randomizing ordering of the destination mapping tableentries.
 12. The method of claim 1, further comprising performing, bythe network device, an initialization, the initialization including therandom selection of the one or more path-selection randomizationoptions.
 13. A network device comprising a non-transitory machinereadable storage medium having instructions stored thereon, wherein theinstructions, when executed by the network device, cause the networkdevice to: randomly select one or more path-selection randomizationoptions to be applied to data packets processed in the network device:receive a data packet; apply the one or more path-selectionrandomization operations to the data packet; determine a next-hop pathfor the data packet based on the one or more path-selectionrandomization operations; and transmit the data packet to a next-hopnetwork device using the determined next-hop path, wherein thepath-selection randomization options comprise operations for modifying ahash key, the operations for modifying a hash key including 1)operations for prepending a first hash key seed to respective hash keysof received data packets; 2) operations for appending a second hash keyseed to the respective hash keys of the received data packets; and 3)operations for masking randomly selected bits of the respective hashkeys of the received data packets.
 14. The network device of claim 13,wherein the instructions, when executed by the network device, furthercause the network device to: determine a hash value for the receiveddata packet based on the selected path-selection randomization options;determine an index for a destination mapping table; and index thedestination mapping table using the determined index to determine thenext-hop path for the data packet.
 15. The network device of claim 14,wherein the instructions for determining the index of the destinationmapping table and indexing the destination mapping table cause thenetwork device to: determine a modulo of the hash value based on anumber of available next-hop paths N and a replication factor rf for acorresponding aggregate of the network device, wherein the destinationmapping table includes rf entries for each respective next-hop path forthe corresponding aggregate.
 16. The network device of claim 15, whereinthe replication factor rf randomly selected for the correspondingaggregate during an initialization of the network device.
 17. Thenetwork device of claim 15, wherein the instructions, when executed,further cause the network device to randomize ordering of thedestination mapping table entries.
 18. A method for processing packetdata in a network device, the method comprising: randomly selecting, bythe network device, one or more path-selection randomization options tobe applied to data packets processed in the network device; receiving,at the network device, a data packet; determining, by the networkdevice, a hash value for the received data packet based on the selectedpath-selection randomization options; determining, based on the hashvalue, an index for a destination mapping table; indexing thedestination mapping table using the determined index to determine thenext-hop path for the data packet; and transmitting, from the networkdevice, the data packet to a next-hop network device using thedetermined next-hop path, wherein the path-selection randomizationoptions comprise operations for modifying a hash key, the operations formodifying a hash key including 1) operations for prepending a first hashkey seed to respective hash keys of received data packets; 2) operationsfor appending a second hash key seed to the respective hash keys of thereceived data packets; and 3) operations for masking randomly selectedbits of the respective hash keys of the received data packets.
 19. Themethod of claim 18, wherein the path-selection randomization optionscomprise: operations for modifying a hash key; a plurality of hashfunctions; and operations for modifying a hash function output value.20. The method of claim 18, wherein randomly selecting one or morepath-selection randomization options includes selecting one of aplurality of hash functions.